Apparatus and method for detecting hovering commands

ABSTRACT

Method and apparatus are provided for detecting hovering commands by a device with a display. In one embodiment, a method for detecting hovering commands by a device with a display comprises detecting a sequence of shadows from a hovering object by one or more light sensors of the device, where the sequence of shadows causes changes in light conditions on the display of the device, determining a leakage current corresponding to changes of light conditions by a controller of the device, determining an action performed by a user based on the changes of light conditions on the display by the controller of the device, determining a hovering command based on the action performed by the controller of the device, and executing the hovering command by the controller of the device.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. non-provisional patentapplication Ser. No. 15/143,303, “Multi-Level Command SensingApparatus,” filed Apr. 29, 2016, which claims the benefit of U.S.provisional patent application No. 62/274,772, “Multi-Level CommandSensing Apparatus,” filed Jan. 4, 2016; both applications are assignedto the assignee hereof. The aforementioned United States patentapplications are hereby incorporated by reference in their entirety.

FIELD

The present invention relates to the field of user interface with adisplay. In particular, the present invention relates to apparatus andmethod for detecting hovering commands.

BACKGROUND

In conventional user interfaces with mobile devices, a user typicallycontrols a mobile device by using one or more fingers to touch or pressa display or keys and buttons of the mobile device. Such touches orpresses are converted into user inputs or commands that may beconfigured to control the mobile device. It is beneficial to allow usersto provide user commands as the user approaches, prior to touching, thescreen of the mobile device. It is also beneficial to allow users toprovide multi-levels of user commands with and/or without touching thescreen of the mobile device.

SUMMARY

Methods and systems are provided for a multi-level command sensingapparatus that includes a matrix of light sensors and light sources andthat may also serve as the display. In one embodiment, a method forperforming multi-level command sensing by a multi-level command sensingapparatus comprises detecting a leakage current corresponding to changesof light conditions on a display by one or more light sensors of themulti-level command sensing apparatus, determining an action performedby a user based on the changes of light conditions on the display by acontroller of the multi-level command sensing apparatus, determining acommand based on the action performed by the controller of themulti-level command sensing apparatus, and executing the command by thecontroller of the multi-level command sensing apparatus.

According to aspects of the present disclosure, when a finger, hand,palm, or object hovers or moves above a display (also referred to as adisplay panel, panel, or screen) or approaches the display before makingdirect contact, the controller can be configured to determine anapproximate position and movement of the finger or object from thechanges in the amount of light detected by the light sensors caused bythe shadow, or shade, cast by the hovering object onto the panel. Thelight sources of the display may be either on or off. The light sensorscan detect less light in the shaded area created by the hovering object.The light sources of the display may be turned on when the amount ofambient light detected is less than a certain predetermined amount.

Methods and systems are provided for a multi-level command sensingapparatus. In one embodiment, a method for determining movement of shadewhile operating under hovering mode includes a light detection sequence.As an object moves over the panel in a 2-dimensional plane parallel tothe panel, the shade caused by the object also moves in certaindirections corresponding to the movement of the object, including leftto right, right to left, top to bottom, or bottom to top. The movementcan also be diagonal or circular. The movement can also be perpendicularto the panel, for example, as an object moves closer to or away from thepanel. The movement can also be combination of the above movements in3-dimensional space. The movements can be classified as 1) movementparallel to the panel 2) movement perpendicular to the panel, and 3) acombination of movement parallel and perpendicular to the panel. Torecognize the movement as an input action to execute a certain command,the system can have a movement interpreter which can compare themovement to a list of predetermined patterns or reference values.

Methods and systems are provided for a multi-level command sensingapparatus. In one embodiment, a method for determining touch includes alight detection sequence. As an object approaches the panel, the shadedarea becomes darker and smaller. When the shaded area becomes smallerthan a predetermined size, smaller than a certain percentage or ratiofrom its initially detected size, darker than a certain percentage orratio of its initially detected darkness, or darker than a predeterminedamount, the panel can be configured to brighten the light sources at theshaded area before the object touches the panel. At the moment of touch,if it occurs, the light sensors can detect a brightening of theinitially shaded area, due to the reflection and scattering of the lightfrom the light sources off the object touching the screen, and thus canbe able to determine when and in what location on the panel the touchinput has occurred. If the brightening does not occur, the object hasapproached the screen to a certain proximity but has not made directcontact.

In another embodiment, a multi-level command sensing apparatus comprisesa display, one or more light sensors configured to detect a leakagecurrent corresponding to changes of light conditions on the display, anda controller comprising one or more processors, where the controller isconfigured to: determine an action performed by a user based on thechanges of light conditions on the display, determine a command based onthe action performed, and execute the command determined.

According to aspects of the present disclosure, the changes of lightconditions comprise a sequence of shadows detected on the display, andthe action performed by the user comprises a sequence of hoveringmotions without touching the display. The changes of light conditionscomprise a sequence of brightened shadows detected on the display, andthe action performed by the user comprises a sequence of touches on thedisplay. The sequence of brightened shadows may be caused by reflectedlight and scattered light from an object touching the display. Thesequence of touches includes a sequence of low pressure touchespredefined by the designer or user. The sequence of touches includes asequence of high pressure touches predefined by the designer or user.The sequence of touches can also include a sequence of multiple levelsof pressure touches predefined by the designer or user.

According to aspects of the present disclosure, the controller may befurther configured to compare the changes of light conditions on thedisplay to a set of predefined changes of light conditions stored in adatabase of the multi-level command sensing apparatus; and identify theaction performed by the user corresponding to the changes of lightconditions on the display in response to a match being found in the setof predefined changes of light conditions.

According to aspects of the present disclosure, the controller may befurther configured to compare the action performed by the user to a setof predefined actions stored in a database of the multi-level commandsensing apparatus, and identify the command corresponding to the actionperformed by the user in response to a match being found in the set ofpredefined actions.

According to aspects of the present disclosure, the controller may befurther configured to authenticate the user based on the sequence oftouches on the display, by continuously authenticating the user duringan access to security sensitive information using the multi-levelcommand sensing apparatus, and terminate the access to the securitysensitive information in response to one or more mismatches found in thecontinuously authenticating process.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned features and advantages of the disclosure, as well asadditional features and advantages thereof, will be more clearlyunderstandable after reading detailed descriptions of embodiments of thedisclosure in conjunction with the non-limiting and non-exhaustiveaspects of following drawings. Like numbers are used throughout thefigures.

FIGS. 1A-1D illustrate methods of detecting changes of light conditionson a display according to aspects of the present disclosure.

FIGS. 2A-2B illustrate methods of detecting changes of light conditionson a display according to aspects of the present disclosure.

FIG. 3A-3B illustrate methods of detecting changes of light conditionsas an object approaches a display according to aspects of the presentdisclosure.

FIGS. 4A-4C illustrate methods of determining actions performed by auser based on the changes of light conditions on the display accordingto aspects of the present disclosure.

FIGS. 5A-5B illustrate methods of detecting changes of light conditionsas an object approaches and touches a display according to aspects ofthe present disclosure.

FIGS. 6A-6D illustrates methods of detecting changes of light conditionsas finger approaches and touches a display according to aspects of thepresent disclosure.

FIGS. 7A-7C illustrate methods of sensing multiple levels of pressure asa display is being pressed according to aspects of the presentdisclosure.

FIGS. 8A-8C illustrate other methods of determining actions performed bya user based on the changes of light conditions on the display accordingto aspects of the present disclosure.

FIGS. 9A-9B illustrates examples of authenticating a user according toaspects of the present disclosure.

FIGS. 10A-10D illustrates other examples of authenticating a useraccording to aspects of the present disclosure.

FIGS. 11A-11D illustrates yet other examples of authenticating a useraccording to aspects of the present disclosure.

FIG. 12 illustrates methods of defining different levels of pressuretouches by a user according to aspects of the present disclosure.

FIG. 13 illustrates an exemplary circuit for detecting a leakage currentcorresponding to changes of light conditions on a display according toaspects of the present disclosure.

FIGS. 14A-14C illustrate examples of OLEDs with light sensors fordetecting a leakage current corresponding to changes of light conditionsaccording to aspects of the present disclosure.

FIG. 15A illustrates an exemplary subpixel circuit cell with forwardbias according to aspects of the present disclosure; FIG. 15Billustrates an exemplary subpixel circuit cell with reverse biasaccording to aspects of the present disclosure.

FIG. 16 illustrates an exemplary pixel circuit cell with RGB subpixelsaccording to aspects of the present disclosure.

FIG. 17 illustrates an exemplary light sensing panel using a thin filmtransistor (TFT) panel structure according to aspects of the presentdisclosure.

FIG. 18 illustrates an exemplary display with light sensors fordetecting changes of light conditions caused by an object hovering abovethe display according to aspects of the present disclosure.

FIG. 19 illustrates an exemplary display with light sensors fordetecting changes of light conditions caused by an object touching thedisplay according to aspects of the present disclosure.

FIG. 20 illustrates an exemplary controller of a multi-level commandsensing apparatus according to aspects of the present disclosure.

FIGS. 21A-21D illustrate methods of performing multi-level commandsensing according to aspects of the present disclosure.

DESCRIPTION OF EMBODIMENTS

Methods and systems are provided for detecting hovering commands by adevice with a display. The following descriptions are presented toenable any person skilled in the art to make and use the disclosure.Descriptions of specific embodiments and applications are provided onlyas examples. Various modifications and combinations of the examplesdescribed herein will be readily apparent to those skilled in the art,and the general principles defined herein may be applied to otherexamples and applications without departing from the scope of thedisclosure. Thus, the present disclosure is not intended to be limitedto the examples described and shown, but is to be accorded the scopeconsistent with the principles and features disclosed herein. The word“exemplary” or “example” is used herein to mean “serving as an example,instance, or illustration.” Any aspect or embodiment described herein as“exemplary” or as an “example” in not necessarily to be construed aspreferred or advantageous over other aspects or embodiments.

Some portions of the detailed description that follows are presented interms of flowcharts, logic blocks, and other symbolic representations ofoperations on information that can be performed on a computer system. Aprocedure, computer-executed step, logic block, process, etc., is hereconceived to be a self-consistent sequence of one or more steps orinstructions leading to a desired result. The steps are those utilizingphysical manipulations of physical quantities. These quantities can takethe form of electrical, magnetic, or radio signals capable of beingstored, transferred, combined, compared, and otherwise manipulated in acomputer system. These signals may be referred to at times as bits,values, elements, symbols, characters, terms, numbers, or the like. Eachstep may be performed by hardware, software, firmware, or combinationsthereof.

According to aspects of the present disclosure, methods and systems areprovided for a touch sensing apparatus. In one embodiment, a method fordetermining touch includes a light detection sequence. As an objectapproaches a display, the shaded area created by the object on the panelbecomes darker and smaller. When the shaded area becomes smaller than apredetermined size, for example smaller than a certain percentage orratio from its initially detected size, darker than a certain percentageor ratio of its initially detected darkness, or darker than apredetermined value, light sources embedded underneath the shaded areain the display will be turned on to detect a touch. A touch may berecognized when one or more light sensors, detect a brightened area inthe initially shaded area, due to the reflection and scattering of thelight from the light sources off the surface of the object touching thescreen. As a result, the touch sensing apparatus can determine when andat what location on the display the touch input has occurred.Alternatively, if the brightened area in the initially shaded area isnot detected, it may be determined that the object has approached thedisplay to certain proximity but has not yet made direct contact withthe display.

In another embodiment, a method for performing multi-level commandsensing in a multi-level command sensing apparatus includes a lightdetection sequence. The light sources are turned on randomly orsequentially. One or more cells or sub pixels turned on at once. As theobject approaches the panel, the shaded area becomes darker and smaller.At the moment of touch, if it occurs, the light sensors can detect abrightening of the initially shaded area, due to the reflection andscattering of the light from the light sources off the object touchingthe screen, and thus can be able to determine when and in what locationon the panel the touch input has occurred. If the brightening does notoccur, the object has approached the screen to certain proximity but hasnot made direct contact.

Once a controller has determined that contact has been made, thecontroller may begin fingerprint capture, using methods described infollowing paragraphs. Based on the area of the image of the fingerprintthat is captured, the controller can be configured to determine therelative pressure applied by the finger. The larger the area of thefingerprint captured, the higher the pressure that has been applied. Thesame procedure can be applied to determine pressure for a stylus with asoft tip, made of material such as rubber.

In another embodiment, an apparatus for determining shade movement,hovering and validity of a fingerprint includes a light refractingdevice (light refractor), a light source, a light collecting device, anda controller. The light refracting device can, for example, be an activematrix organic light emitting diode (AMOLED) panel structure withreverse current measurement and amplification circuitry, and include animaging surface and a viewing plane. Incident light from the lightsource is projected directly or indirectly onto the imaging surface tocreate an image of the patterned object from the projected light ontothe viewing plane. The apparatus is configured to have a thin formfactor, which may be flexible or conformable, compared to conventionaloptical fingerprint acquisition apparatuses. The AMOLED panel includesthe light source panel as well as light collecting devices. Themulti-level command sensing apparatus can be implemented as an in-cellstructure.

In another embodiment, an apparatus for determining shade movement,hovering and validity of a fingerprint includes a light refractingdevice (light refractor), a light source, a light collecting device, anda controller. The light refracting device can, for example, be athin-film transistor (TFT) panel and include an imaging surface, a lightreceiving surface, a viewing plane, and light collecting devices. Thelight source can be an individually addressable panel of discrete lightsources, for example, a liquid crystal display (LCD) panel or an AMOLEDpanel. Incident light from the light source is projected through thelight receiving surface and projected directly or indirectly onto theimaging surface to create an image of the patterned object from theprojected light onto the viewing plane. The apparatus is configured tohave a thin form factor, which may be flexible or conformable, comparedto conventional optical fingerprint acquisition apparatuses. The TFTpanel may be implemented as an add-on panel that is placed on top of thelight source panel.

In another embodiment, a method for determining pressure or size of afingerprint includes detecting the number of pixels or sub pixels thatare brightened. The brightened pixels, also referred to as brightenedshadows, are detected by determining which pixels have a higher leakagecurrent than a predetermined value or a predetermined percentage orratio of its initial value. The larger the number of brightened pixels,the higher the pressure that has been applied. In one embodiment,certain numbers of brightened pixels correspond to certain pressures insome fixed relationship determined in advance. In another embodiment,the number of brightened pixels is compared to the number of brightenedpixels initially detected when the panel is touched. A larger number ofbrightened pixels than the number of brightened pixels initiallydetected corresponds to a higher pressure. In another embodiment, thenumber of brightened pixels is compared to some reference touch input bythe user, calibrated in advance. A larger number of brightened pixelsthan the number of brightened pixels determined in the reference touchcorrespond to a higher pressure. In another embodiment, the number ofbrightened pixels is compared to touch input data collected by thedevice as the device is used over time. A certain characteristic of thedata, such as its average, or some other reference determined byprocessing the data using various methods may be used. A larger numberof brightened pixels than the number determined by processing the datacorresponds to a higher pressure. A smaller number of brightened pixelsthan the number determined by processing the data corresponds to asmaller pressure.

In another embodiment, a method for determining size and validity of afingerprint includes determining a set of light sources from a pluralityof light sources for emitting light to a fingerprint, determining a setof sensor zones from a plurality of sensor zones for sensing scatteredlight from the fingerprint, determining a minimum distance between asensor zone and a light source for sensing selective scattered lightfrom the fingerprint, emitting light from the set of light sources togenerate the scattered light from the fingerprint, sensing the scatteredlight in the set of sensor zones, and determining a validity of thefingerprint using the scattered light sensed in the plurality of sensorzones.

The light refracting device supports basic touch screen functionality,as well as hovering object detection and gesture recognition, andpressure level detection using a combination of ambient light and theactive matrix display as light sources.

FIGS. 1A-1B illustrate a method of detecting changes of light conditionson a display with the display being turned off according to aspects ofthe present disclosure. In the example of FIG. 1A, with a finger 102 orobject hovering above the panel 106 at a distance larger than 0 butclose enough to cast a detectable shadow, for example approximatelybetween 30 millimeters (mm) to 1 mm, the presence of the finger 102 orobject creates a shadow 104 on the panel due to the presence of ambientlight 108 in the background. The shadow 104 on the panel 106 produces adifference in the leakage current between the shadowed area and thebright area measured by a matrix of light sensors embedded in the panel106. The apparatus can then determine the position of the finger 102 orobject based on the location of the shadow 104.

An object hovering farther away from the panel may produce a smallerdifference in leakage current between the shadowed area it creates onthe panel and the brighter area outside the shadowed area, than thedifference produced with the object hovering closer to the panel. Thepanel may be configured to detect a hovering object if the difference inleakage current between the shadowed area and the brighter area aroundthe shadow is larger than a certain value that may be determined inadvance or calculated based on present conditions.

Multiple surrounding objects may cast both wanted and unwanted shadowsonto the panel. The panel may be configured to detect multiple shadowsor only detect a certain number of the darkest shadows cast onto thepanel.

Similarly in FIG. 1B, the hovering object 102 creates a shadow 105 onthe panel 106 and a lower leakage current at the shadowed area relativeto the area around the shadow. As the object 102 approaches closer tothe panel 106 (also referred to as the screen or the display), theshadow 105 becomes darker and creates an even lower leakage current thanthe example as in FIG. 1A.

FIGS. 1C-1D illustrate a method of detecting changes of light conditionson a display with the display being turned on according to aspects ofthe present disclosure. As shown in FIG. 1C, the panel 106 is turned on,which emits light 110 from the panel 106. With a finger 102 or objecthovering above the panel 106 at a distance approximately between 30 mmto 1 mm, the presence of the finger 102 or object creates a shadow 107on the display panel due to the presence of ambient light 108 in thebackground and the emitted light 110. The shadow 107 on the panel 106produces a difference in the leakage current between the shadowed areaand the bright area measured by a matrix of light sensors embedded inthe panel 106. The apparatus can then determine the position of thefinger 102 or object based on the location of the shadow 107.

Similarly in FIG. 1D, the hovering object 102 creates a shadow 109 onthe panel 106 and a lower leakage current at the shadowed area relativeto the area around the shadow. As the object 102 approaches closer tothe panel 106 (also referred to as the screen or the display), theshadow 109 becomes darker and creates an even lower leakage current thanthe example as in FIG. 1C.

According to aspects of the present disclosure, the touch panel mayinclude a light source panel, which may also be a display screen, alongwith a matrix of light sensors. The matrix of light sensors detect andlocate the position of a finger or stylus placed on the panel to providetouch screen functionality on par with or surpassing other methods.

A controller can be configured to detect the movement of hoveringobjects by detecting significant changes in amount of light detectedeither throughout the panel, or limited to a certain region. The changesof light conditions can be used to determine the position and/ormovement of the hovering object.

FIG. 2A illustrates a method of detecting changes of light conditionswith the display being turned on, and FIG. 2B illustrates a method ofdetecting changes of light conditions with the display being turned offaccording to aspects of the present disclosure. The examples of FIGS.2A-2B illustrate a finger or object 202 hovering near but not yetdirectly touches the display 206. In the presence of ambient light 208,the leakage current detected by light sensors in the exposed area 210,labeled as area B, may be larger than the leakage current detected inthe shaded area 204, labeled as area A, by the finger 202 or object. InFIG. 2A, the display 206 may be lit by one or more illumination lights207 from one or more light sources. In FIG. 2B, the display 206 may beturned off, i.e. not lit by one or more illumination lights, and themethod of detecting changes of light conditions with the display maystill be implemented.

FIG. 3A illustrates a method of detecting changes of light conditions asan object approaches a display with the display being turned on, andFIG. 3B illustrates a method of detecting changes of light conditions asan object approaches a display with the display being turned offaccording to aspects of the present disclosure. FIGS. 3A-3B illustratethe changes a controller may observe as the finger or object 302approaches the panel. As the finger or object gets closer, for examplemoving from a distance of 25 mm to 5 mm, to the screen 306, the shadowof the finger or object on the screen becomes darker and darker (D→F),causing less leakage current at the shadowed area. The leakage currentdetected at region D 303 is larger than the leakage current detected atregion E 304, which is larger than the leakage current detected atregion F 305. The sensor can detect the leakage current variation andchange to “touching” mode when the shadow becomes smaller than apredetermined size, or produces a leakage current smaller than apredetermined amount. In FIG. 3A, with the display is turned on, witharrows 310 represent light being emitted from panel 306.

In some implementations, such as the embodiment shown in FIG. 3B, evenwith the panel light source off, with the presence of ambient light, acontroller can be configured to detect hovering objects and thus executecommands even with the display off. For example, swiping a hand abovethe screen from one side to another (left to right or right to left) cancause the matrix of sensors to sense a change in ambient light beingdetected from one side of the screen to the other. This pattern can beutilized in a music player application to correspond to changing a songto either the next or the previous song, depending on the direction ofthe swipe. Swiping a hand above the screen from the top of the screen tothe bottom of the screen, or from the bottom of the screen to the top ofthe screen can be programmed to correspond to adjusting the volume up ordown.

FIGS. 4A-4C illustrate methods of determining actions performed by auser based on the changes of light conditions on the display accordingto aspects of the present disclosure. FIGS. 4A-4C illustrate examples ofdifferent gestures that may be produced by a hovering finger or object402. A controller can be configured to recognize the gestures by themovement of the shadow (for example 404, 406, 408, or 410) on the panel,which produces differences in leakage current measured by the lightsensors. Gesture recognition can be performed while hovering or touchingthe screen. Various gestures or a combination of gestures can beprogrammed to represent various commands. Detection of a stationaryhovering object may also be configured to correspond to various commandsdepending on the situation. For example, detection of a hovering objectover a link or button on the screen for a certain amount of time may beconfigured to open a secondary menu related to the link or button.

According to aspects of the present disclosure, specific gestures oractions in a certain sequence, such as swiping above the screen fromleft to right, right to left, top to bottom, bottom to top, left toright to left, right to left to right, top to bottom to top, bottom totop to bottom, a direct touch motion gesture followed by pressuredetection, circling motion, etc., or some and any combination of theabove can be used to correspond to various commands during general useor to certain actions or directions in a game.

FIG. 5A illustrates a method of detecting changes of light conditions asan object approaches and touches a display with the display being turnedon, and FIG. 5B illustrates a method of detecting changes of lightconditions as an object approaches and touches a display with thedisplay being turned off according to aspects of the present disclosure.FIG. 5A illustrates the differences in leakage current that may beobserved by a controller when a finger or object 502 makes directcontact with the panel 504 when the display is on. FIG. 5B illustratesthe differences in the leakage current that may be observed by acontroller when a finger or object 502 makes direct contact with thepanel 504 when the display is off.

According to aspects of the present disclosure, when the display is off,the device may be in “standby” mode and ready to accept input to wake upthe display, after which the device may be unlocked according to varioususer verification methods. With the presence of ambient light, thematrix of sensors may be able to detect the presence of a finger on thescreen for a pre-designated period of time, after which the display canbe woken up to read and verify a fingerprint. Other methods may also beused to initially detect the presence of a finger on the screen for thedesignated period of time—for example, a pressure sensor can be used.Alternatively, other methods can be used to initially signal the deviceto wake up in order to turn on the display and read a fingerprint—forexample, pressing the home or power button, or double tapping thescreen.

FIG. 6A illustrates a method of detecting changes of light conditions asa finger or other object 602 approaches a display 604 before makingdirect contact, with the display turned off; FIG. 6B illustrates amethod of detecting changes of light conditions as a finger or otherobject 602 approaches and touches a display 604 with the display turnedoff according to aspects of the present disclosure. FIG. 6C illustratesa method of detecting changes of light conditions as finger or otherobject 602 approaches a display 604 before making direct contact; FIG.6D illustrates a method of detecting changes of light conditions as afinger or other object 602 approaches and touches a display 604 with thedisplay being turned on according to aspects of the present disclosure.

In the examples of FIG. 6B and FIG. 6D, when the finger or object makescontact with the panel, the area at which it makes contact may bebrighter than before due to the light generated by the screen reflectingoff the finger or object back onto the panel. The light sensors thusmeasure a higher leakage current at the area of contact.

FIGS. 7A-7C illustrate a method of sensing multiple levels of pressureas a finger or other object touches a display according to aspects ofthe present disclosure. In FIGS. 7A-7C, the size of the scanned image ofthe finger or object 702 on the panel 704 changes in correspondence tothe changes of pressure of the finger or object on the screen. Thepressure of the finger can be determined by the area of touch capturedby a controller. FIG. 7A illustrates a simple touch, in which thecontroller scans a small area of the fingerprint 706. FIG. 7Billustrates a slight higher pressure relative to the simple touchapplied to the screen, in which the controller scans a largerfingerprint area 708. FIG. 7C illustrates an even higher relativepressure applied to the screen, in which the controller scans an evenlarger fingerprint area 710.

According to aspects of the present disclosure, a controller may beconfigured to provide pressure sensing capabilities based on the area oftouch captured. The area of touch may be compared to a predeterminedreference value, which may be an absolute default value, the initialvalue detected upon initial contact, a value determined throughcalibration through input by the user, or some reference valuedetermined through other methods. The area of touch in relation to thereference value may be used to determine the level of pressure exertedby the finger or other object on the screen. A smaller area of touchrelative to the reference value can be configured to correspond to alighter pressure, and a larger area of touch relative to the referencevalue can be configured to correspond to a higher pressure. Any numberof different discrete levels of pressure may be designated in comparisonto the reference value, or varying levels of pressure may be determinedon a continuous scale.

According to aspects of the present disclosure, the multiple levels ofpressure being applied to a display may be customized on an individualuser basis. A user may train the controller of the multi-level commandsensing device to learn the levels of light pressure touch, mediumpressure touch, and high pressure touch, such as the different levels ofpressure being applied and their corresponding different sizes of areasas shown in FIG. 7A, FIG. 7B, and FIG. 7C, respectively. The pressuredetection functionality is also compatible with stylus pens with a softtip, such as rubber. Pressure can be detected in a rapid tap, a pressinghold, or continuously while the finger or stylus moves across thescreen.

FIGS. 8A-8C illustrate other methods of determining actions performed bya user based on the changes of light conditions on the display accordingto aspects of the present disclosure. In the examples of FIGS. 8A-8C,paths of a finger or an object (for example 802, 804, 806, 808, 810,812, 814, or 816) can be traced by a finger on the panel. The changes inlight conditions in the areas on the panel may be used to determine thelocation of the finger or the object.

In some implementations, with the display on, and with or withoutambient light, swiping a hand across the screen (from left to right orright to left) can cause the matrix of sensors to sense a change inlight being detected from one side of the screen to the other. Thisgesture can be programmed as commands that correspond to examining thenext or previous item while browsing through some list of items. Thegesture can also be programmed as commands along with a direct touchswipe to represent another level of browsing through items or data. Forexample, the hovering swipe gesture may flip to the next page of items,while the direct touch swipe switches to the next item on the same page.

FIGS. 9A-9B illustrates examples of authenticating a user according toaspects of the present disclosure. In the example of FIG. 9A, a user maydesignate a specific area 902 on a display 904 for unlocking the device.A default area may also be designated. The device may only be unlockedif the finger is placed on or within the correct pre-designated area,and the validity of the fingerprint is subsequently verified.

According to aspects of the present disclosure, a controller may beconfigured to provide fingerprint recognition capabilities. Fingerprintsmay be read anywhere on the screen and in any orientation, or thefunctionality may be enabled at a certain region or regions of thescreen. Multiple fingerprints may also be read simultaneously. Thecontroller may also support simultaneous and/or continuous fingerprintrecognition during the use of the device, in which it captures andidentifies fingerprints while the finger is moving on the screen orwhile the user is typing.

In some implementations, a combination of methods for unlocking thedevice may be used and intuitively integrated—for example, after wakingup the display, a pattern may be drawn, and the finger may remain on thescreen without lifting for a short period of time after drawing thepattern, so that the device may verify the validity of the fingerprint.This can provide an additional layer of security for the device,requiring both a correct pattern and a valid fingerprint. This method ofdouble verification can be seamlessly executed with a panel that servesboth as a display and a fingerprint sensor, as it does not require twoseparate actions—one of drawing the pattern on the screen, and one ofscanning the fingerprint on a separate fingerprint sensor—on the part ofthe user.

Since the fingerprint recognition technology is integrated into theentire screen and not limited to a certain area or areas on the device,a fingerprint may be read for verification to unlock the device anywhereon the display screen, in any position or orientation. In someimplementations, a certain area may be specifically designated forfingerprint capture for unlocking the device by the user. Additionally,in some other implementations, multiple fingerprints or a palm print canbe read and identified simultaneously by the controller and be requiredto unlock the device, providing additional security.

In the example of FIG. 9B, the apparatus may be configured tocontinuously authenticate the user during an access to securitysensitive information. In one exemplary implementation, the apparatusmay be configured to authenticate the user when the user touches one ormore links on a display 904, such as Link A 906 _(A) to Link L 906 _(L),or when the user presses one or more buttons, such as button 908 ₁ tobutton 908 _(N), or when the user types using a displayed keyboard 910.Upon detecting one or more mismatches found in the continuouslyauthenticating process, the apparatus may terminate the access to thesecurity sensitive information.

According to aspects of the present disclosure, the controller canprovide an additional layer of security for sensitive mobileapplications after the phone itself has been unlocked by requiring thatthe fingerprint be read and verified in the moment that the icon for aparticular mobile application is selected. The fingerprint may bedirectly scanned on the screen in the moment of or after making aselection without having to be lifted and placed on another area on thedevice. Access may be granted if the fingerprint is consistent with thatof the owner of the device or otherwise determined to be valid. Inaddition, the device may recognize fingerprints on the screen directlyduring active operation of device for sensitive data or material, suchas a bank account. The device may be configured to lock if a foreignfingerprint or fingerprints are detected during the operation. An exactnumber or threshold may be specified.

FIGS. 10A-10D illustrates other examples of authenticating a useraccording to aspects of the present disclosure. In the examples of FIGS.10A-10D, the verification of a fingerprint may be performed after firstdrawing a pattern to unlock the device. After waking the display, apattern is drawn (for example, 1004, 1006, 1008, 1010 or 1012), and thefinger remains on the screen without lifting for a short period of time,allowing the device to verify the validity of the fingerprint (forexample, 1014 a, 1014 b, 1014 c, 1014 d, 1014 e, or 1014 f). The devicemay be configured to unlock only after both the correct pattern is drawnand the validity of the fingerprint is verified.

FIGS. 11A-11D illustrate other examples of authenticating a useraccording to aspects of the present disclosure. In the examples of FIGS.11A-11D, fingerprint recognition may be performed during active use ofthe touchscreen. The controller can be configured to capturefingerprints, such as 1102 a, 1102 b, 1102 c, 1102 d, or 1102 e, on thescreen during active operation of the device for sensitive information,such as a bank account. The device may be configured to lock if anunauthenticated fingerprint or fingerprints are detected. An exactnumber or threshold may be specified.

FIG. 12 illustrates examples of various pressure-based touch commandsaccording to aspects of the present disclosure. Different commands, suchas 1202 to 1232, may be designated based on different levels ofpressures of touches in addition to different types of gestures/motionson the screen. In some embodiments, the different commands correspondingto different levels of pressure being applied to a display may becustomized by a user of the multi-level command sensing device. The usermay train the controller of the multi-level command sensing device tolearn the different individualized pressures of touches, such as thedifferent areas of up, down, left, right, diagonal, circular, zigzagmotions, etc. as shown by numerals 1202 to 1232 in FIG. 12.

FIG. 13 illustrates an exemplary circuit for detecting a leakage currentcorresponding to changes in light conditions on a display according toaspects of the present disclosure. In FIG. 13, an exemplary controlcircuit for determining whether an object or finger is hovering ortouching the panel and its corresponding position is shown. Theexemplary control circuit 1300 comprises a leakage current detectionphase, a comparison phase, and a command generation phase. In oneparticular implementation, the exemplary control circuit 1300 includesone or more pixel sampling blocks 1302, one or more pixel leakagecurrent detectors 1304, one or more current converters 1306, a pluralityof comparators 1308, a plurality of registers 1310, one or more codematching blocks 1312, a plurality of AND blocks 1314, and one or morecommand generators 1316. The exemplary control circuit 1300 furtherincludes a register divider generator 1320, an average generator 1322, areference leakage current generator 1324, a counter 1326, a generatorcomparator 1328, and a command selection block 1330. The abovecomponents are communicatively coupled as shown in FIG. 13 to form theexemplary control circuit 1300.

According to aspects of the present disclosure, the control circuit canbe configured to compare the leakage currents of each position in thematrix of sensors to a reference. Based on reference leakage currentsthat come from experimental data, the controller can determine thedifference between hovering and touching modes on the screen. Referenceleakage voltage for hovering can be smaller than the minimum leakagevoltage of touching mode. Each comparator detects the pixels' leakagecurrents, which is converted to voltage and compared to a resistordivider voltage generator. It generates a command which represents theaction on the screen. In some implementations, a set of user definedcommands may be generated and stored in a database or memory of themulti-level command sensing apparatus. If output data from comparatorsmatch one of the existing commands in the memory, it selects thecorresponding action from the controller through the command generator.The output of command generator selects one of the expected actions. Inone exemplary implementation, the processing can be synchronized using asynchronization clock.

FIGS. 14A-14C illustrate examples of OLEDs with light sensors fordetecting a leakage current corresponding to changes of light conditionsaccording to aspects of the present disclosure. FIG. 14A illustrates aunidirectional OLED 1402 with light sensors; FIG. 14B illustrates aconformable OLED 1404 with light sensors; and FIG. 14C illustrates abi-directional OLED 1406 with light sensors according to aspects of thepresent disclosure.

According to aspects of the present disclosure, both top emission andbottom emission type OLED structures can be used as the main componentof a fingerprint acquisition apparatus. Several different types of OLEDdevices, such as small molecule OLED, polymer OLED, or solution basedOLED, may be utilized as main OLED device structures. Both transparentand non-transparent OLED panels can be used as the main component of afingerprint acquisition apparatus. Both thin panel and flexible orconformable types of OLED panels can be used as the main component of afingerprint acquisition apparatus.

An active matrix OLED (AMOLED) panel can be used as the main componentof a fingerprint acquisition apparatus. An AMOLED panel may includesubpixel areas (red, green, and blue subpixels) and a driving circuitarea (thin film transistor and capacitor). The brightness of eachsubpixel can be adjusted by the driving and switching transistors andcapacitors and by controlling the amount of current injected to the OLEDsubpixels. The dimension of subpixels can be formed using OLED materialdeposition techniques. For instance, the size and position of subpixelscan be set by using shadow masks during the OLED material evaporationprocess.

An OLED may have a layered structure with the following sequence:anode/hole injection layer/hole transport layer/emissive layer/electrontransport layer/electron injection layer/cathode. ITO and othertransparent conducting materials having high work function can be usedfor anode materials, and metals such as aluminum and magnesium can beused for cathode materials. In some implementations, the imaging surfacewould be at the bottom of the substrate, and the light emission planewould be the cathode layer. The optical structure may include thetransparent layers between the substrate and the cathode.

FIG. 15A illustrates an exemplary subpixel circuit cell with forwardbias according to aspects of the present disclosure; FIG. 15Billustrates an exemplary subpixel circuit cell with reverse biasaccording to aspects of the present disclosure.

The reliability of such a fingerprint acquisition apparatus, i.e. theOLED panel lifetime, can be improved by using various sealing techniquesand materials, such as desiccant, frit glass sealing, and thin filmencapsulation. Various types of substrates such as sapphire, glass, andplastic materials can be used for OLED carriers in order to control thelight travel path (refractive index control), to enhance/improve signalto noise ratio of image sensing, and to improve the reliability andlifetime of fingerprint apparatus. FIG. 15A shows an exemplary AMOLEDsubpixel unit cell circuit (2D-driving TFT circuit with subpixels). Thedriving area may include a driving transistor, switching transistor,holding capacitor, and reverse current sensor. FIG. 15B shows thereverse current read and amplified in the OLED circuit structure.

FIG. 16 illustrates an exemplary pixel circuit cell with RGB subpixelsaccording to aspects of the present disclosure. In some embodiments, anAMOLED panel has a three-subpixel structure. In the subpixel structure,for example, a blue subpixel can be used as a light source while theneighboring green or red subpixels may be used as a sensor because theband gap of blue subpixels is larger than that of the green or redsubpixels. FIG. 16 shows an exemplary R/G/B pixel structure where theblue subpixel is the light source, and the green or red subpixel is thesensor. The reverse voltage can be biased in the sensor subpixel whenthe lighting subpixel is turned on. In FIG. 15B, the I-V curvescorrespond with subpixel structures in FIG. 16. The amount of reversecurrent in the sensor subpixel under reverse bias is increased whenlight is reflected, refracted, or scattered from a fingerprint to thesensor subpixel. The amount of reverse current can be measured usingcurrent sensing circuits in the driving circuit area. The reversecurrent signal can be amplified using an amplification circuit, and/or asignal processor. The amplified current signal can then be processed togenerate a fingerprint image by a signal processing algorithm.

The OLED panel resolution can be controlled by varying the size anddensity of each subpixel and by setting the subpixel structure of theOLED panel. For example, an OLED panel may have a larger lightingcomponent (e.g. blue subpixels) and a smaller sensor component (e.g.green and/or red subpixels). According to aspects of the presentdisclosure, subpixel structures can have different sizes. The subpixeldensity can be enhanced by changing pixel shape from stripe type tocircular or diamond shape. In addition, an OLED subpixel structure canhave different shapes, such as square, rectangle, circle, diamond, etc.The patterning of the subpixel structure can be fabricated by using finemetal mask processes, ink-jet printing, or laser transfer technologies.

FIG. 17 illustrates an exemplary light sensing panel using a thin filmtransistor (TFT) panel structure according to aspects of the presentdisclosure. Each cell of the TFT panel structure can be an addressablelight sensing component, referred to as a sensing pixel. In the exampleshown in FIG. 17, capture sensor 1700 includes a passivation layer 1718,which can be formed of SiNx. On top of passivation layer 1718, a storagecapacitor layer is formed including first electrode 1715. This storagecapacitor layer is preferably formed from indium tin oxide (ITO), whichis conductive and transparent. On top of first electrode 1715, aninsulating layer 1717 is formed, preferably of SiNx. Over insulatinglayer 1717, a second electrode 1714 is formed, preferably of tin oxide.First electrode 1715, insulating layer 1717 and second electrode 1714together form the storage capacitor. Over second electrode 1714, anotherinsulating layer 1716 is formed, which can be formed from SiNx. A layerof glass layer 1711 is placed over insulating layer 1716. A fingerprintto be imaged is placed on glass layer 1711, which may be referred toherein as the imaging surface.

A light sensing unit 1712 (also referred to as a light sensor), which ispreferably a thin-film transistor, and a switching unit 1713, which isalso preferably a thin-film transistor, are horizontally arranged on apassivation layer 1718. Under passivation layer 1718, a back light 1720irradiates light upward to be passed through the fingerprint capturesensor 1700. As shown in FIG. 17, back light 1720 can be separated froma lower, exposed surface of passivation layer 1718. It is alsoconsidered, however, that backlight 1720 be placed against lower surfaceof passivation layer 1718. Backlight 1720 can be an LED or any othertype of light source. A source electrode 1712-S of the light sensingunit 1712 and a drain electrode 1713-D of the switching unit 1713 areelectrically connected through second electrode 1714. A gate electrode1712-G of the light sensing unit 1712 is connected to first electrode1715. Additionally, a first light shielding layer 1713-sh is placedbetween insulating layer 1717 and passivation layer 1718 at switchingunit 1713. As detailed below, first light shielding layer 1713-sh blockslight from backlight 1720 from reaching switching unit 1713.Additionally, second light shielding layer 1722 is positioned betweenglass layer 1711 and insulating layer 1716 at switching unit 1713 toshield switching unit 1713 from light passing through or reflected fromglass layer 1711.

In the above structure, a photosensitive layer 1712-P such as amorphoussilicon (a-Si:H) is formed between the drain electrode 1712-D and sourceelectrode 1712-S of the light sensing unit 1712. Note thatphotosensitive layer 1712-P allows current to flow in response to apredetermined amount of light striking a surface of photosensitive layer1712-P. In this way, when more than a predetermined quantity of light isreceived at a surface of photosensitive layer 1712-P, current flowsthrough the drain electrode 1712-D and the source electrode 1712-S.

According to aspects of the present disclosure, in a method offabricating capture sensor 1700, a second light shielding layer 1722 isfirst placed on glass layer 1711 via evaporation, sputtering or anyother method. Glass layer 1711 is preferably between about 5 and 10 um,though may be either thicker or thinner. Light shielding layer 1722 ispreferably formed from a metal such as aluminum, but may be formed fromany suitable light blocking material. Next, insulating layer 1716 isformed on top of glass layer 1711 and second light shielding layer 1722.As noted above, insulating layer 1716 is preferably formed from SiNx.Photosensitive layer 1712-P is then formed over insulating layer 1716.As discussed above, photosensitive layer 1712-P is preferably formedfrom a-Si:H. Source electrode 1712-D of light sensing unit 1712, secondelectrode 1714 and drain electrode 1713-D of switching unit 1713 arenext formed over insulating layer 1716. Source electrode 1712-D, secondelectrode 1714 and drain electrode 1713-D are each preferably formed ofITO, but may be formed of any suitable conductor. Next, insulating layer1717 is formed and over insulating layer 1717 first electrode 1715 isformed. Insulating layer 1717 is preferably formed from SiNx and firstelectrode 1715 is preferably formed of ITO but may be formed of anysuitable conductor. Next, gate electrode 1712-G of light sensing unit1712 and light shield 1713-sh are formed. Preferably, gate electrode1712-G and light shielding layer 1713-sh are each formed of ITO, but maybe formed of any suitable material and light shielding layer 1713-shdoes not need to be formed from the same material as gate electrode1712-G. Next, passivation layer 1718, which is preferably formed fromSiNx, is formed over first electrode 1715, gate electrode 1712-G andlight shielding layer 1713-sh. As discussed above, backlight 1720 caneither be attached to the lower, exposed surface of passivation layer1718 or separately supported.

In another implementation, an image capture sensor can havesubstantially the same structure as capture sensor shown in FIG. 17except that conductive ITO layer is placed beneath glass layer and aninsulating layer, which can be formed of SiNx, is placed below ITOlayer. Because ITO layer is conductive, electrostatic charge built up onglass layer can be discharged by connecting ITO layer to a ground. Thiscan prevent damage to capture sensor. Image capture sensor can befabricated in substantially the same manner as image capture sensorexcept that ITO layer is formed over glass layer and insulating layer isformed over ITO layer prior to forming light shielding layer overinsulating layer.

In yet another implementation, an image capture sensor can havesubstantially the same structure as capture sensor shown in FIG. 17.Specifically, the capture sensor includes a light sensing unit, which issubstantially the same and light sensing unit, and switching unit, whichis substantially the same as switching unit, formed between aninsulating layer and a passivation layer. However, above insulatinglayer capture sensor includes a substrate layer having a plurality offiber-optic strands running in a direction perpendicular to a surface ofsubstrate layer. Preferably, the diameter of the fiber-optic strands 330a forming substrate layer is from about 4 um to about 8 um in diameterand more preferably about 6 um in diameters, though larger or smallerdiameters can also be used. Substrate layer can be formed from glassfiber optic strands 330 a or fiber optic strands of other substantiallytransparent materials including polymers. Fiber optic sheets can be usedto form the substrate layer.

FIG. 18 illustrates an exemplary display with light sensors fordetecting changes of light conditions caused by an object 1802 hoveringabove the display according to aspects of the present disclosure. FIG.19 illustrates an exemplary display with light sensors for detectingchanges of light conditions caused by an object 1902 touching thedisplay according to aspects of the present disclosure.

A light sensor panel can be implemented as an add-on panel that isplaced on top of a light source panel. The light source panel can be,for example, an LCD panel or an AMOLED panel. FIG. 18 illustrates a TFTtype light sensor panel 1804 is placed on top of an LCD display panelstructure 1806 as an add-on panel. Similarly, FIG. 19 illustrates a TFTtype light sensor panel 1904 is placed on top of an LCD display panelstructure 1906 as an add-on panel. In the examples of FIG. 18 and FIG.19, a TFT type light sensor panel is placed on top of an LCD panelstructure as an add-on panel. The sensing pixels of the TFT type lightsensing panel can be individually addressable and can be activatedaccording to a designated sensor zone pattern.

If there are non-transparent areas in the light sensor panel, theseareas can be aligned with the non-transparent areas of the light sourcepanel. As an example, TFT light sensor panel may be aligned with an LCDpanel structure, wherein non-transparent components of the TFT lightsensor panel are aligned with the black matrix areas of the LCD displaypanel structure. In this approach, the TFT light sensor panel is alignedwith the LCD panel structure. The non-transparent components on the TFTlight sensor panel are aligned with the black matrix area on the LCDdisplay panel structure.

The black matrix areas of the LCD display panel are non-transparent andtherefore would block the transmission of the display backlight. Thelight sensor panel can be designed so that its non-transparent areas canbe aligned with the black matrix areas of the LCD panel. When the LCDdisplay emits light through the transparent areas of the LCD display,this light can be used as the light source for the light sensor panel.The LCD display can individually control cells (individuallyaddressable) to emit light as discrete light sources that are projectedinto the light refractor according to a designated illumination pattern.

As described above, the light refracting device can, for example, alsobe a thin-film transistor (TFT) add-on panel placed on top of an LCD orAMOLED display panel structure that acts as a panel of light sources.Incident light from the light source panel is projected through thelight receiving surface and projected directly or indirectly onto theimaging surface to create an image of the patterned object from theprojected light onto the viewing plane. This multi-level command sensingapparatus can be also used as a touch sensor when implemented in amobile device.

FIG. 20 illustrates an exemplary controller of a multi-level commandsensing apparatus according to aspects of the present disclosure. Asshown in FIG. 20, a controller 2000 of the multi-level command sensingapparatus may include one or more processors 2002, a network interface2004, a database 2006, a multi-level command sensing engine, 2008, amemory 2010, and a user interface 2012. In some implementations, themulti-level command sensing apparatus may be a part of a mobile device.

According to aspects of the present disclosure, a mobile device isusually equipped with a touch sensor. If a mobile device was equippedwith the multi-level command sensing apparatus of the presentdisclosure, then the touch sensor would not be required, as themulti-level command sensing apparatus may also be used as a touchsensor. As described herein, a mobile device can be configured toinclude a multi-level command sensing apparatus for fingerprintrecognition. In some implementations, the mobile device may comprise awireless transceiver which is capable of transmitting and receivingwireless signals via wireless antenna over a wireless communicationnetwork. Wireless transceiver may be connected to a bus by a wirelesstransceiver bus interface. The wireless transceiver bus interface may,in some embodiments be at least partially integrated with wirelesstransceiver. Some embodiments may include multiple wireless transceiversand wireless antennas to enable transmitting and/or receiving signalsaccording to a corresponding multiple wireless communication standardssuch as, for example, versions of IEEE Std. 802.11, CDMA, WCDMA, LTE,UMTS, GSM, AMPS, Zigbee and Bluetooth®, etc.

The mobile device may also comprise a SPS receiver capable of receivingand acquiring SPS signals via a SPS antenna. The SPS receiver may alsoprocess, in whole or in part, acquired SPS signals for estimating alocation of the mobile device. In some embodiments, processor(s),memory, DSP(s) and/or specialized processors (not shown) may also beutilized to process acquired SPS signals, in whole or in part, and/orcalculate an estimated location of the mobile device, in conjunctionwith the SPS receiver. Storage of SPS or other signals for use inperforming positioning operations may be performed in memory orregisters (not shown).

In addition, the mobile device may comprise digital signal processor(s)(DSP(s)) connected to the bus by a bus interface, processor(s) connectedto the bus by a bus interface and memory. The bus interface may beintegrated with the DSP(s), processor(s) and memory. In variousembodiments, functions may be performed in response execution of one ormore machine-readable instructions stored in memory such as on acomputer-readable storage medium, such as RAM, ROM, FLASH, or discdrive, just to name a few example. The one or more instructions may beexecutable by processor(s), specialized processors, or DSP(s). Thememory may comprise a non-transitory processor-readable memory and/or acomputer-readable memory that stores software code (programming code,instructions, etc.) that are executable by processor(s) and/or DSP(s) toperform functions described herein. In a particular implementation, thewireless transceiver may communicate with processor(s) and/or DSP(s)through the bus to enable the mobile device to be configured as awireless station as discussed above. Processor(s) and/or DSP(s) mayexecute instructions to execute one or more aspects of processes/methodsdiscussed above in connection with FIGS. 21A-21D.

According to aspects of the present disclosure, a user interface maycomprise any one of several devices such as, for example, a speaker,microphone, display device, vibration device, keyboard, touch screen,etc. In a particular implementation, the user interface may enable auser to interact with one or more applications hosted on the mobiledevice. For example, devices of user interface may store analog ordigital signals on the memory to be further processed by DSP(s) orprocessor in response to action from a user. Similarly, applicationshosted on the mobile device may store analog or digital signals on thememory to present an output signal to a user. In another implementation,the mobile device may optionally include a dedicated audio input/output(I/O) device comprising, for example, a dedicated speaker, microphone,digital to analog circuitry, analog to digital circuitry, amplifiersand/or gain control. In another implementation, the mobile device maycomprise touch sensors responsive to touching or pressure on a keyboardor touch screen device.

The mobile device may also comprise a dedicated camera device forcapturing still or moving imagery. The dedicated camera device maycomprise, for example an imaging sensor (e.g., charge coupled device orCMOS imager), lens, analog to digital circuitry, frame buffers, etc. Inone implementation, additional processing, conditioning, encoding orcompression of signals representing captured images may be performed atthe processor(s) or DSP(s). Alternatively, a dedicated video processormay perform conditioning, encoding, compression or manipulation ofsignals representing captured images. Additionally, the dedicated videoprocessor may decode/decompress stored image data for presentation on adisplay device on the mobile device.

The mobile device may also comprise sensors coupled to the bus which mayinclude, for example, inertial sensors and environment sensors. Inertialsensors may comprise, for example accelerometers (e.g., collectivelyresponding to acceleration of the mobile device in three dimensions),one or more gyroscopes or one or more magnetometers (e.g., to supportone or more compass applications). Environment sensors of the mobiledevice may comprise, for example, temperature sensors, barometricpressure sensors, ambient light sensors, and camera imagers,microphones, just to name few examples. The sensors may generate analogor digital signals that may be stored in memory and processed by DPS(s)or processor(s) in support of one or more applications such as, forexample, applications directed to positioning or navigation operations.

In a particular implementation, the mobile device may comprise adedicated modem processor capable of performing baseband processing ofsignals received and down-converted at a wireless transceiver or SPSreceiver. Similarly, the dedicated modem processor may perform basebandprocessing of signals to be up-converted for transmission by thewireless transceiver. In alternative implementations, instead of havinga dedicated modem processor, baseband processing may be performed by aprocessor or DSP (e.g., processor(s) or DSP(s)).

FIG. 21A illustrates method of performing multi-level command sensingaccording to aspects of the present disclosure. In the example shown inFIG. 21A, in block 2102, the method detects a leakage currentcorresponding to changes of light conditions on a display by one or morelight sensors of a multi-level command sensing apparatus. In block 2104,the method determines an action performed by a user based on the changesof light conditions on the display by a controller of the multi-levelcommand sensing apparatus. In block 2106, the method determines acommand based on the action performed by the controller of themulti-level command sensing apparatus. In block 2108, the methodexecutes the command by the controller of the multi-level commandsensing apparatus.

According to aspects of the present disclosure, the changes of lightconditions may comprise a sequence of shadows detected on the display,and the action performed by the user may comprise a sequence of hoveringmotions without touching the display. The changes of light conditionsmay also comprise a sequence of brightened shadows detected on thedisplay, and the action performed by the user may comprise a sequence oftouches on the display. The sequence of brightened shadows can be causedby reflected light and scattered light from an object touching thedisplay. The sequence of touches may include a sequence of low pressuretouches predefined by the user. The sequence of touches may include asequence of high pressure touches predefined by the user.

FIG. 21B illustrates a method of determining an action performed by auser based on changes of light conditions on a display according toaspects of the present disclosure. As shown in FIG. 21B, in block 2112,the method compares the changes of light conditions on the display to aset of predefined changes of light conditions stored in a database ofthe multi-level command sensing apparatus. In block 2114, the methodidentifies the action performed by the user corresponding to the changesof light conditions on the display in response to a match being found inthe set of predefined changes of light conditions.

FIG. 21C illustrates determining a command based on the action performedby the user according to aspects of the present disclosure. As shown inFIG. 21C, in block 2122, the method compares the action performed by theuser to a set of predefined actions stored in a database of themulti-level command sensing apparatus. In block 2124, the methodidentifies the command corresponding to the action performed by the userin response to a match being found in the set of predefined actions.

FIG. 21D illustrates a method of authenticating a user based on asequence of touches on the display by the user according to aspects ofthe present disclosure. As shown in FIG. 21D, in block 2132, the methodauthenticates the user based on the sequence of touches on the display.Optionally (indicated by dotted lines) or additionally, the method maycontinuously authenticate the user during an access to securitysensitive information using the multi-level command sensing apparatus,and terminate the access to the security sensitive information inresponse to one or more mismatches found in the continuouslyauthenticating process.

It will be appreciated that the above descriptions for clarity havedescribed embodiments of the invention with reference to differentfunctional units and processors. However, it will be apparent that anysuitable distribution of functionality between different functionalunits or processors may be used without detracting from the invention.For example, functionality illustrated to be performed by separateprocessors or controllers may be performed by the same processors orcontrollers. Hence, references to specific functional units are to beseen as references to suitable means for providing the describedfunctionality rather than indicative of a strict logical or physicalstructure or organization.

The invention can be implemented in any suitable form, includinghardware, software, firmware, or any combination of these. The inventionmay optionally be implemented partly as computer software running on oneor more data processors and/or digital signal processors. The elementsand components of an embodiment of the invention may be physically,functionally, and logically implemented in any suitable way. Indeed, thefunctionality may be implemented in a single unit, in a plurality ofunits, or as part of other functional units. As such, the invention maybe implemented in a single unit or may be physically and functionallydistributed between different units and processors.

One skilled in the relevant art will recognize that many possiblemodifications and combinations of the disclosed embodiments may be used,while still employing the same basic underlying mechanisms andmethodologies. The foregoing description, for purposes of explanation,has been written with references to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described to explain the principles of theinvention and their practical applications, and to enable others skilledin the art to best utilize the invention and various embodiments withvarious modifications as suited to the particular use contemplated.

We claim:
 1. A method of detecting hovering commands by a device with anactive matrix organic light emitting diode (AMOLED) display, comprising:detecting a sequence of shadows from a hovering object by one or morelight sensors of the AMOLED display, wherein the sequence of shadowscauses changes in light conditions on the AMOLED display, wherein thechanges in light condition are caused by shadows of the object formed,by ambient light in the background and by emitted light of the AMOLEDdisplay; determining a leakage current corresponding to changes of lightconditions by a controller of the device, wherein the leakage currentdecreases as the object approaches the AMOLED display, wherein theAMOLED display includes a plurality of pixel circuits, each pixelcircuit includes a blue subpixel circuit, a green subpixel circuit, anda red subpixel circuit; and wherein the green subpixel circuit and thered subpixel circuit are configured to sense the changes of lightconditions, the blue subpixel circuit is configured to generate theemitted light of the AMOLED display in response to an ambient light inan environment of the device being less than a predetermined amount;detecting, by the green subpixel circuit and the red subpixel circuit ineach pixel circuit in the plurality of pixel circuits, the ambient lightin an environment of the device being less than the predeterminedamount; emitting, by the blue subpixel circuit in each pixel circuit inthe plurality of pixel circuits, a light to the hovering object from theAMOLED display of the device; and receiving, by the green subpixelcircuit and the red subpixel circuit in each pixel circuit in theplurality of pixel circuits, a reflected light from the hovering object,wherein the reflected light causes the changes in light conditions onthe AMOLED display; determining an action performed by a user based onthe changes of light conditions on the AMOLED display by the controllerof the device; determining a hovering command based on the actionperformed by the controller of the device; and executing the hoveringcommand by the controller of the device.
 2. The method of claim 1:wherein the action performed by the user comprises a sequence ofhovering motions without touching the AMOLED display, and wherein thehovering object has a distance of 1 mm to 30 mm from the AMOLED display.3. The method of claim 1: wherein the action comprises a parallelmovement of the hovering object relative to a surface of the AMOLEDdisplay.
 4. The method of claim 1: wherein the action comprises aperpendicular movement of the hovering object relative to a surface ofthe AMOLED display.
 5. The method of claim 1: wherein the actioncomprises an angled movement of the hovering object relative to asurface of the AMOLED display.
 6. The method of claim 1: wherein theaction comprises a circular movement of the hovering object relative toa surface of the AMOLED display.
 7. The method of claim 1, whereindetermining the action performed by the user comprises: comparing thechanges of light conditions on the AMOLED display to a set of predefinedchanges of light conditions stored in a database of the device; andidentifying the action performed by the user corresponding to thechanges of light conditions on the AMOLED display in response to a matchbeing found in the set of predefined changes of light conditions.
 8. Themethod of claim 7, wherein comparing the changes of light conditions onthe AMOLED display comprises: converting the leakage current to aleakage voltage; comparing the leakage voltage to a set of referencevoltages; and storing comparison information in one or more registers ofthe controller.
 9. The method of claim 1, wherein determining thecommand comprises: comparing the action performed by the user to a setof predefined actions stored in a database of the device; andidentifying the command corresponding to the action performed by theuser in response to a match being found in the set of predefinedactions.
 10. A device for detecting hovering commands, comprising: anactive matrix organic light emitting diode (AMOLED) display; one or morelight sensors configured to detect a sequence of shadows from a hoveringobject by one or more light sensors of the AMOLED display, wherein thesequence of shadows causes changes in light conditions on the AMOLEDdisplay, wherein the chances in light condition are caused by shadows ofthe object formed, by ambient light in the background and by emittedlight of the AMOLED display; a controller comprising one or moreprocessors, the controller is configured to: determine a leakage currentcorresponding to changes of light conditions, wherein the leakagecurrent decreases as the object approaches the AMOLED display, whereinthe AMOLED display includes a plurality of pixel circuits, each pixelcircuit includes a blue subpixel circuit, a green subpixel circuit, anda red subpixel circuit; and wherein the green subpixel circuit and thered subpixel circuit are configured to sense the changes of lightconditions, the blue subpixel circuit is configured to generate theemitted light of the AMOLED display in response to ambient light in anenvironment of the device being less than a predetermined amount;determine an action performed by a user based on the changes of lightconditions on the AMOLED display; determine a hovering command based onthe action performed; and execute the hovering command; wherein thecontroller is further configured to: detect, by the green subpixelcircuit and the red subpixel circuit in each pixel circuit in theplurality of pixel circuits, the ambient light in an environment of thedevice being less than the predetermined amount; emit, by the bluesubpixel circuit in each pixel circuit in the plurality of pixelcircuits, a light to the hovering object from the AMOLED display of thedevice; and receive, by the green subpixel circuit and the red subpixelcircuit in each pixel circuit in the plurality of pixel circuits, areflected light from the hovering object, wherein the reflected lightcauses the changes in light conditions on the AMOLED display.
 11. Thedevice of claim 10: wherein the action performed by the user comprises asequence of hovering motions without touching the AMOLED display, andwherein the hovering object has a distance of 1 mm to 30 mm from theAMOLED display.
 12. The device of claim 10: wherein the action comprisesa parallel movement of the hovering object relative to a surface of theAMOLED display.
 13. The device of claim 10: wherein the action comprisesa perpendicular movement of the hovering object relative to a surface ofthe AMOLED display.
 14. The device of claim 10: wherein the actioncomprises an angled movement of the hovering object relative to asurface of the AMOLED display.
 15. The device of claim 10: wherein theaction comprises a circular movement of the hovering object relative toa surface of the AMOLED display.
 16. The device of claim 10, wherein thecontroller is further configured to: compare the changes of lightconditions on the AMOLED display to a set of predefined changes of lightconditions stored in a database of the device; and identify the actionperformed by the user corresponding to the changes of light conditionson the AMOLED display in response to a match being found in the set ofpredefined changes of light conditions.
 17. The device of claim 16,wherein the controller is further configured to: convert the leakagecurrent to a leakage voltage; compare the leakage voltage to a set ofreference voltages; and store comparison information in one or moreregisters of the controller.
 18. The device of claim 10, wherein thecontroller is further configured to: compare the action performed by theuser to a set of predefined actions stored in a database of the device;and identify the command corresponding to the action performed by theuser in response to a match being found in the set of predefinedactions.